There are commercial felt caps no need to cut up hats. You want a bigger cap as you have to cover what is there.

4

Speaker Parts, Dust Caps, Gaskets, Lead Wire, My Audio Addiction

Google "speaker repair parts" to find suppliers. That guy has 3.75, MEASURE FIRST, I always used 4's. You want enough room to work without touching the aluminum.

BE CAREFUL SUBSTITUTING GLUES! A quality speaker glue is best. You might want to talk to a local reconer and get his advice.